notebook.community

Edit and run



In [1]:

    
cd /Users/Will/Documents/GITHUB/class_project/class_project/HW7









    



/Users/Will/Documents/GITHUB/class_project/class_project/HW7



In [13]:

    
from data_cleaning_utils import *
import matplotlib.pyplot as plt
%matplotlib inline

This is a quick notebook designed to show the functions within the data_cleaning_utils.py There are 3 main functions:

data import tool
data smoothing tool
resampling tool

right now, each function is written to run individually, but can be modified so that they aren't re-run each time



In [3]:

    
cd /Users/Will/Documents/GITHUB/class_project/class_project/Data/Amazon









    



/Users/Will/Documents/GITHUB/class_project/class_project/Data/Amazon



In [4]:

    
raw = import_data()









    



datetime column name? datetime



In [5]:

    
raw.head()









    Out[5]:






  
    
      
      date
      time
      datetime
      battery
      fdom_rfu
      fdom_qsu
      ODO_percent
      ODO_mg_L
      Temp
      Cond_us_cm
      pH
    
    
      datetime
      
      
      
      
      
      
      
      
      
      
      
    
  
  
    
      2014-11-01 22:35:00
      11/1/14
      22:35:07
      2014-11-01 22:35:00
      6.1
      -4.67
      20.91
      96.8
      7.29
      30.172
      52
      7.04
    
    
      2014-11-01 22:35:00
      11/1/14
      22:35:08
      2014-11-01 22:35:00
      6.1
      -4.65
      20.97
      96.8
      7.29
      30.172
      52
      7.04
    
    
      2014-11-01 22:35:00
      11/1/14
      22:35:09
      2014-11-01 22:35:00
      6.1
      -4.63
      21.02
      96.8
      7.29
      30.172
      52
      7.05
    
    
      2014-11-01 22:35:00
      11/1/14
      22:35:10
      2014-11-01 22:35:00
      6.1
      -4.62
      21.07
      96.8
      7.29
      30.172
      52
      7.05
    
    
      2014-11-01 22:35:00
      11/1/14
      22:35:11
      2014-11-01 22:35:00
      6.1
      -4.61
      21.08
      96.7
      7.29
      30.172
      52
      7.05



In [16]:

    
smooth = smooth_data("pH")









    



datetime column name? datetime
What size windows do you want for the moving average? 45



In [9]:

    
reduced = reducer()









    



datetime column name? datetime
What size windows do you want for the moving average? 45
What frequency would you like to resample to? Format = XS(seconds), XT(minutes)3T
3258772
14384



In [10]:

    
reduced.head()









    Out[10]:






  
    
      
      battery
      fdom_rfu
      fdom_qsu
      ODO_percent
      ODO_mg_L
      Temp
      Cond_us_cm
      pH
    
    
      datetime
      
      
      
      
      
      
      
      
    
  
  
    
      2014-11-01 22:33:00
      6.1
      -2.716415
      26.768868
      95.909434
      7.227925
      30.169962
      52.000000
      7.040815
    
    
      2014-11-01 22:36:00
      6.1
      -1.824833
      29.442611
      95.642222
      7.208611
      30.160289
      52.000000
      7.044889
    
    
      2014-11-01 22:39:00
      6.1
      -1.629944
      30.027056
      95.163333
      7.176056
      30.133572
      52.000000
      7.050537
    
    
      2014-11-01 22:42:00
      6.1
      -3.172333
      25.399500
      94.136111
      7.098222
      30.138600
      52.416667
      7.051852
    
    
      2014-11-01 22:45:00
      6.1
      -2.010500
      28.885222
      93.510556
      7.053944
      30.116406
      53.000000
      7.049222

	date	time	datetime	battery	fdom_rfu	fdom_qsu	ODO_percent	ODO_mg_L	Temp	Cond_us_cm	pH
datetime
2014-11-01 22:35:00	11/1/14	22:35:07	2014-11-01 22:35:00	6.1	-4.67	20.91	96.8	7.29	30.172	52	7.04
2014-11-01 22:35:00	11/1/14	22:35:08	2014-11-01 22:35:00	6.1	-4.65	20.97	96.8	7.29	30.172	52	7.04
2014-11-01 22:35:00	11/1/14	22:35:09	2014-11-01 22:35:00	6.1	-4.63	21.02	96.8	7.29	30.172	52	7.05
2014-11-01 22:35:00	11/1/14	22:35:10	2014-11-01 22:35:00	6.1	-4.62	21.07	96.8	7.29	30.172	52	7.05
2014-11-01 22:35:00	11/1/14	22:35:11	2014-11-01 22:35:00	6.1	-4.61	21.08	96.7	7.29	30.172	52	7.05

	battery	fdom_rfu	fdom_qsu	ODO_percent	ODO_mg_L	Temp	Cond_us_cm	pH
datetime
2014-11-01 22:33:00	6.1	-2.716415	26.768868	95.909434	7.227925	30.169962	52.000000	7.040815
2014-11-01 22:36:00	6.1	-1.824833	29.442611	95.642222	7.208611	30.160289	52.000000	7.044889
2014-11-01 22:39:00	6.1	-1.629944	30.027056	95.163333	7.176056	30.133572	52.000000	7.050537
2014-11-01 22:42:00	6.1	-3.172333	25.399500	94.136111	7.098222	30.138600	52.416667	7.051852
2014-11-01 22:45:00	6.1	-2.010500	28.885222	93.510556	7.053944	30.116406	53.000000	7.049222